The Fast Bilateral Solver


The Bilater Solver is a novel algorithm for edge-aware smoothing that combines the flexibility and speed of simple filtering approaches with the accuracy of domain-specific optimization algorithms. This algorithm was presented by Jonathan T. Barron and Ben Poole as an ECCV2016 oral and best paper nominee. Algorithm details and applications can be found in https://arxiv.org/pdf/1511.03296.pdf .
(This moudle will be contributed to OpenCV_contrib soon)



Introduce


Algorithm

Implementation

SBS
The two bilateral representations used in the paper, here shown filtering a toy one-dimensional grayscale image of a step-edge. This toy image corresponds to a 2D space visualized here (x = pixel location, y = pixel value) while in the paper we use RGB images, which corresponds to a 5D space (XYRGB). The lattice (Fig 2a) uses barycen-tric interpolation to map pixels to vertices and requires d+1 blurring operations, where d is the dimensionality of the space. The simplified bilateral grid (Fig 2b) uses nearest-neighbor interpolation and requires d blurring operations which are summed rather than done in sequence. The grid is cheaper to construct and to use than the lattice, but the use of hard assignments means that the filtered output often has blocky piecewise-constant artifacts.
Slice:

void FastBilateralSolverFilterImpl::Splat(Eigen::VectorXf& input, Eigen::VectorXf& output)
{
output.setZero();
for (int i = 0; i < int(splat_idx.size()); i++)
{
output(splat_idx[i]) += input(i);
}
}

Blur:

void FastBilateralSolverFilterImpl::Blur(Eigen::VectorXf& input, Eigen::VectorXf& output)
{
output.setZero();
output = input * 10;
for (int i = 0; i < int(blur_idx.size()); i++)
{
output(blur_idx[i].first) += input(blur_idx[i].second);
}
}

Slice:

void FastBilateralSolverFilterImpl::Slice(Eigen::VectorXf& input, Eigen::VectorXf& output)
{
output.setZero();
for (int i = 0; i < int(splat_idx.size()); i++)
{
output(i) = input(splat_idx[i]);
}
}
// solve Ay = b
Eigen::ConjugateGradient<Eigen::SparseMatrix<float>, Eigen::Lower|Eigen::Upper> cg;
cg.compute(A);
cg.setMaxIterations(bs_param.cg_maxiter);
cg.setTolerance(bs_param.cg_tol);
// y = cg.solve(b);
y = cg.solveWithGuess(b,y0);
std::cout << "#iterations: " << cg.iterations() << std::endl;
std::cout << "estimated error: " << cg.error() << std::endl;

Reference

article{BarronPoole2016,
author = {Jonathan T Barron and Ben Poole},
title = {The Fast Bilateral Solver},
journal = {ECCV},
year = {2016},
}
@article{Barron2015A,
author = {Jonathan T Barron and Andrew Adams and YiChang Shih and Carlos Hern\'andez},
title = {Fast Bilateral-Space Stereo for Synthetic Defocus},
journal = {CVPR},
year = {2015},
}
@article{Adams2010,
author = {Andrew Adams Jongmin Baek Abe Davis},
title = {Fast High-Dimensional Filtering Using the Permutohedral Lattice},
journal = {Eurographics},
year = {2010},
}

Installation Instructions


Build OpenCV

This is just a suggestion on how to build OpenCV 3.3. There a plenty of options. Also some packages might be optional.

sudo apt-get install libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev python-dev python-numpy libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libjasper-dev libdc1394-22-dev
git clone https://github.com/opencv/opencv.git
git clone https://github.com/opencv/opencv_contrib.git
cd opencv
mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D WITH_TBB=ON -D BUILD_NEW_PYTHON_SUPPORT=ON -D WITH_V4L=ON -D INSTALL_C_EXAMPLES=ON -D INSTALL_PYTHON_EXAMPLES=ON -D BUILD_EXAMPLES=ON -D WITH_QT=ON -D WITH_OPENGL=ON -D BUILD_TIFF=ON -DBUILD_opencv_viz=OFF -DWITH_VTK=OFF WITH_EIGEN=ON -D BUILD_EXAMPLES=ON -DOPENCV_EXTRA_MODULES_PATH=<opencv_contrib>/modules ..
make -j
sudo make install

Remember change opencv_contrib to your own opencv_contrib path. This will create two demos that include in OpenCV executable examples, that you can run as shown in below.


Try Demos


Colorize

colorize
colorize video(click here to watch the vedio)

#in you opencv build folder
./bin/example_ximgproc_colorize YOUR/OWN/PICTURE --dst_path=PATH/YOU/WANT/TO/SAVE/RESULT

rose1

rose2

rose3

rose4

Depthsuperresolution

depth
depth demo(click here to watch the vedio)

target
the target.

result
the result of fast bilateral solver.

Basic Usage


Depthsuperresolution:

cv::ximgproc::fastBilateralSolverFilter(reference,target,confidence,result,
spatialSigma,lumaSigma,chromaSigma,100);

Firstly, we use the reference image mat_R construct a BilateralGrid, the we launch a depthsuperresolution to optimize the target image mat_T. The parameter spatialSigma is the Gaussian kernal for coordinate x y, similarly , the lumaSigma correspond luma(Y) and the chromaSigma correspond chroma(UV). It need to be noted that he mat_R should be covert to YUV form before construct the bilateralgrid.

Colorization

cv::cvtColor(target, target, cv::COLOR_BGR2YCrCb);
std::vector<cv::Mat> src_channels;
std::vector<cv::Mat> dst_channels;
cv::split(target,src_channels);
cv::Mat result1 = cv::Mat(mat_input_gray.size(),mat_input_gray.type());
cv::Mat result2 = cv::Mat(mat_input_gray.size(),mat_input_gray.type());
filtering_time = (double)getTickCount();
dst_channels.push_back(src_channels[0]);
cv::ximgproc::fastBilateralSolverFilter(src_channels[0],src_channels[1],mat_input_confidence,
result1,sigma_spatial,sigma_luma,sigma_chroma);
dst_channels.push_back(result1);
cv::ximgproc::fastBilateralSolverFilter(src_channels[0],src_channels[2],mat_input_confidence,
result2,sigma_spatial,sigma_luma,sigma_chroma);
dst_channels.push_back(result2);
cv::merge(dst_channels,target);
cv::cvtColor(target, target, cv::COLOR_YCrCb2BGR);

Similar to above, we need to covert the imput image mat_in(gray image for colorization) to YUV form, then draw the gray image. when the drawing finished, press “ESC” twice to launch the colorization procession. the result will be save in specified folder.

FastBilaterSolverFilter

Ptr< FastBilateralSolverFilter> fbs = createFastBilateralSolverFilter(guide,
sigma_spatial, sigma_luma, sigma_chroma, num_iter, max_tol);
fbs->filter(src, confidence, dst);

Similar to BilateralGrid, the PermutohedralLattce also need spatial parameter and the color parameter to specified the Gaussian kernel.


Schedule


Item State Remark
C++ code of the core algorithm Completed also python
Depthsuperres module Completed optimized
Colorization module Completed now in opencv example
PermutohedralLatticeFilter Completed will not contribe to opencv
Contribute project to OpenCV Completed with sample and test file
Detail Documentation Completed this toturial